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Introduction 



Since 1980, the Center for Development of Human Services (CDHS) 
has provided high quality competency-based training (CBT) to so- 
cial services agencies throughout New York State. These programs 
were based on needs identified in different ways including profes- 
sional judgment, focus groups, and comprehensive, structured sur- 
veys. 

During this time, CDHS developed a competency-based training 
management system which is "an integrated assembly of interact- 
ing elements designed to carry out cooperatively a predetermined 
function" (Gibson, 1960). For the past 20 years, CDHS used the 
system to manage program activities for a large university-based 
training organization that specializes in public-sector, social ser- 
vices training, primarily in New York State. 

The concept of general systems theory was introduced in 1937 by 
Ludwig von Bertalaffy, a biologist. He concluded that mathemati- 
cal models in biological, behavioral, and social sciences have similar 
structures and that based on these similarities, it was possible to 
develop a general theory to unify different scientific disciplines 
(Bertalaffy, 1968). Systems theory, therefore, is a set of interre- 
lated principles that explains and predicts the behavior of people, 
groups, or organizations. A well-designed system is an orderly 
whole that clearly shows "the interrelationships of parts to each 
other and to the whole itself (Silvern, 1965, p. 1). 

By relating common principles of different systems, systems theory 
improves communication and provides a holistic scientific approach. 
The major functions of systems theory are to: 

investigate the isomorphy of concepts, laws, and models in 
various fields; 

help make useful transfers from one field to another; 

encourage development of adequate theoretical models in the 
fields which lack them; 
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Introduction Continued 



minimize duplication of theoretical, effort in different fields; 

promote the unity of science by improving communication 
among specialists from different fields (Bertalanffy, 1968, p. 
15). 

Mesarovic (1964) noted that systems theory must be sufficiently 
abstract to encompass different types of specialized theories with 
concepts and terms defined in context at the appropriate LeveL of 
abstraction. It must describe common features, but avoid specific 
aspects of particular systems. The system must clearly communi- 
cate how goals are achieved and describe interrelationships that 
exist among goals, procedures, and outcomes. A clear statement 
of systems theory clarifies assumptions about human behavior that 
underlie a program. It describes the measures needed to anaLyze 
program outcomes and establishes an iterative cycle of develop- 
ment, testing, and redesign (Scheirer, 1994). 

In the early 1960s, systems theory dominated program pLanning and 
budgeting in industry and defense (Schlesinger, 1963; WiLdavsky, 
1964). It also exerted a significant influence on education and 
training. However, the term was not wideLy used untiL the space 
program popularized it with the words, "All systems are go!" At the 
present time, systems models exert a significant influence on train- 
ing and education. 

The CDHS modeL possesses the characteristics of a true system which 
focuses on the most important aspects of training: 

What should trainees Learn that wiLL improve their job perfor- 
mance? 

Have the trainees mastered the competencies on which 
training was based? 

How did training contribute to improved organizational 
performance? 
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The system is a compLete model that begins with statewide needs 
assessments by job title with data collected from local agencies. 
Staff use these data to develop specific, needs-driven curricula. It 
includes computerized procedures to schedule training and produce 
evaluation reports at the state, regional, and local levels using a 
competency- based criterion-referenced evaluation design (CREST) 
(McCowan, McGregor, & LoTempio, S., 1983; 1984). The system in- 
cludes comprehensive follow-up procedures to analyze transfer of 
learning at worker and agency level, and it provides remedial train- 
ing for trainees who fail to master important attitudes, skills, and 
knowledge covered during training. 

Competency-based training (CBT) must have the following charac- 
teristics: 

Clear job descriptions and program outcomes 

Needs assessments based on job-related competencies 

Structured hierarchy of domains, competencies, and objectives 

Posttest assessment of trainee performance compared with 
clear criteria 

Remedial training and OJT mentoring to assure trainee mastery 
of essential material 

Many programs, several of which are nationally prominent and well- 
regarded, are incorrectly promoted as competency-based training 
because they do not assess trainee performance. These programs 
have clear goals and objectives and curriculums linked to these 
objectives, but they are not competency-based if they do not 
assess trainee performance and provide supplementary instruction 
for persons who fail to master essential attitudes, skills, or knowl- 
edge. TMS is a complete system that collects evaluation data at 
the reaction, learning, behavior and results levels described by 
Kirkpatrick (1994) (see Table 1). 
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Table 1 

Four Levels of Evaluation 

Level 1 Evaluate how well trainees Liked a program, typicaLLy using a 

Reaction guestionnaire. Base items on job relevance, training effectiveness, 
and trainer performance. 

Level 2 Examine how well trainees mastered knowledge and skills 

Learning incLuded in training objectives. TypicaLLy, Learning is measured by 
multiple-choice items. 

Level 3 Evaluate program-reLated on-the-job performance. Assess change 

Behavior reLated to training objectives using guestionnaires, interviews, 
or direct observation. 

Level 4 ReLate training to organizational objectives (e.g., reduction 

Results in costs or absenteeism; increased productivity or quality). 
Calculate cost/benefit or ROI (return on investment). 

Level 4 assessment, which is based on results related to organiza- 
tions objectives (e.g., cost-benefit, return on investment, or worker 
performance on the job), is seldom conducted, particularly in the 
public sector, primarily because of cost and accessibility to data. 
CDHS has conducted level 4 results-based studies. For example, a 
major study involves an examination of reports completed by a 
sample of approximately 400 New York City risk assessment work- 
ers who completed a 3-day training program conducted by CDHS. 
Copies of records completed by these workers will be evaluated by 
multiple raters, and these ratings will be compared to the posttest 
performance and supervisor ratings of the same workers. This pro- 
cess will determine if the posttest has predictive validity relative 
to on-the-job performance. 
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Phases of Training Management System 

The CDHS training management system begins with job analysis as 
state and local agency staff collaboratively identify competencies 
that relate significantly to effective job performance. Successive 
phases flow from these competencies as training and assessment 
packages are developed and training programs are conducted. Dur- 
ing formative evaluation, training practices are modified using data 
on trainee achievement and trainer performance. The complete pro- 
cess is illustrated in Figure 1, while specific procedures for each 
phase are detailed in Figures 2, 3, 4, and 5. 

Figure 1 
Mission 

New York State Department of Social Services 
Office of Human Resource Development 

Training Management System 




Figure 2 

Mission Phase of MTS 
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Mission Phase Continued 



The mission of an organization describes its basic societal func- 
tion in terms of products and services provided for its clientele. The 
mission of CDHS provides the organization with a foundation for 
developing large-scale programs that provide high quality training 
for New York State human service workers. 

As illustrated in Figure 2, the Mission phase includes goal selec- 
tion, job analysis, competence development, and needs assessment. 
The process, which progressively moves from higher order, more 
general concepts to statements of greater specificity, is straight- 
forward and logical. During the first step, management selects train- 
ing goals congruent with the broad mission of the department. A 
goal, quite simply, is the purpose toward which an endeavor is 
directed. These goals are reviewed formally and informally by other 
managers, supervisors, and staff. Goals are modified and refined un- 
til a consensus is reached regarding their appropriateness. This pro- 
cess is conducted periodically to provide assurances that stated 
goals meet current, ongoing needs. 

Job analysis begins with an examination of tasks involved in spe- 
cific job titles based on Civil Service job descriptions and functional 
responsibilities. Descriptions for similar job titles from other agen- 
cies and organizations are collected and examined. Concurrently, 
supervisors study staff to identify behaviors that relate to compe- 
tent performance. Information is collected from other sources in- 
cluding published research, program evaluations, staff and client 
focus groups, and staff performance assessments. This pool of in- 
formation is used to generate a competency-based job description 
that is reviewed and validated by staff and supervisors. Their sug- 
gestions are incorporated in an edited version that is recycled 
through the review process. 

Finally, a panel of experts review and modify the job description 
until it accurately describes competencies required for the job. For 
the purposes of training, the job description is based on compe- 
tencies which are related clusters of specific attitudes, skills, and 
knowledge. These competencies are the basis for all subsequent 
training activities. 
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Mission Phase Continued 



Needs assessment focuses on the gaps in performance which exist 
between competent and ineffective staff. Identified competencies 
are used as Likert items in the needs assessment. Individual staff 
members are evaluated by themselves and their supervisors on each 
competence based on level of skill and the relevance of each com- 
petence. Evaluators produce individual training needs assessments 
and group profiles from these needs assessment. This information 
is used to develop the training curriculum. The needs assessment 
phase includes preliminary activities related to training, such as 
selection of key staff and contractors and selection of potential 
trainee populations. 
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Program Design 

Figure 3 

Program Design Phase of MTS 




Trainer Preparation 


Curriculum Design 


Test Design 


Test Analysis 


Describe trainer 
tasks and activities. 


Design a 

curriculum model. 


Describe the 
purpose of the test. 


Field-test 

instrument. 


List desired 
trainer competencies. 


Specify 

learning conditions. 


Form a test 

development 

committee 


Obtain feedback 
after field-test. 


Prepare trainer 
job descriptions. 


Specify types 
of learning. 


Describe the 
population 
to be tested. 


Determine 

reliability. 


Recruit 

qualified trainers. 


Describe format 
of test items. 


Develop format 
for objectives. 


Compute index 
of item 

discrimination. 


Assess competence 
and background. 


Train staff to 
write objectives. 


Estimate 
the number 
of items required. 


Compute 
item difficulty. 


Provide trainers 
with background 
information. 


Write objectives 
for competences. 


Develop 
an initial pool 
of item. 


Determine 
mastery levels. 


Brief trainers 
on program goals. 


Sequence objectives. 


Match items 
to competencies. 


Revise test. 


Provide practice 
opportunities. 


Identify 

training activities. 


Add items 
when appropriate. 


Administer 
revised test. 


Develop 

trainer readiness. 


Adapt for 
adult trainee 
characteristics. 


Complete 
item editing. 


Reassess test. 


Design a peer 
coaching program. 


Develop 

training prototype. 


Establish 
content validity. 


Complete 
final revisions. 


Observe 

trainer performance. 


Field-test 
prototype 
and products. 


Write instructions 
for scoring and 
administration. 




Provide enrichment 
opportunities. 


Assess performance 
of field-test subjects. 


Prepare table shells. 






Modify training 
prototype. 


Prepare draft 
of instrument. 





Program Design Phase Continued 



The Program Design phase includes trainer and trainee preparation, 
curriculum design, test design, and test analysis. Specific proce- 
dures for each of these steps are contained in Figure 3. 

Trainer preparation begins when program managers describe trainer 
responsibilities. Consistent with the concept of competency-based 
training, the job description is based on competencies required for 
highly successful training. Much of the success achieved by the sys- 
tem is a direct result of recruiting highly qualified staff. Trainers 
have excellent academic credentials, as well as substantial profes- 
sional experience directly related to job-related topics presented 
during training. Strong interpersonal and presentation skills are es- 
sential, as well as organizational support through supervision, train- 
ing, and enrichment opportunities. 

During curriculum design, goal-related competencies are refined into 
a more detailed level of specificity when training objectives (i.e., 
attitudes, skills, and knowledge) are prepared. This is a critical step 
because all training and evaluation materials relate to, and flow 
from, these objectives. 

The level of specificity (i.e., unit of analysis) required for a train- 
ing objective is an important issue which requires greater elabo- 
ration than can be provided in this monograph. However, the most 
important point related to this concept involves the job level for 
which training is designed. For beginning workers, training objec- 
tives are very specific and focus on entry level skills. For trainees 
at the supervisory and management levels, objectives become pro- 
gressively more general and inclusive. Cognitive and educational 
psychologists have discussed this concept from different perspec- 
tives including "chunking," (Mislevy, 1993) "chaining," and "asso- 
ciation" (Gagne & Briggs, 1974). Essentially, the concept implies 
that people learn complex tasks by mastering a series of specific 
skills required to perform the task. Once a complex task is perfected, 
it becomes a specific skill required to perform even more complex 
behaviors. For example, first grade children must recognize letters 
before they learn to read. As they mature, they master increasingly 
more complex reading skills. In a similar manner, the behaviors re- 
quired for entry level workers differ in complexity from those re- 
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Program Design Phase Continued 



quired for supervisors. Likewise, the behaviors required of supervi- 
sors are less complex than those required for management level 
staff. As job level increases, competencies move to higher cogni- 
tive levels of analysis, synthesis, and evaluation. 

A competency-based curriculum contains the entire range of directed 
experiences completed by trainees. Curriculum activities are highly 
structured and flow directly from the competencies and training 
objectives. Management and staff select the objectives on which 
training will be based, since in most circumstances training will 
not include every objective from the total universe of those avail- 
able. As noted earlier, a competence is a cluster of related objec- 
tives, and each objective is a specific attitude, skill, or knowledge. 
Since the number of specific objectives is usually extensive, train- 
ers select and sequence objectives on which training will be based 
and develop a prototype curriculum. 

This monograph does not discuss techniques for writing specific 
instructional objectives because these have been described in other 
publications (Eisner, 1969; Gronlund, 1970; Mager, 1962; McCowan 
& Wegenast, 1996; Popham, 1969; 1990). However, protocols for 
writing objectives must be followed carefully because objectives 
form the foundation for all curriculum and evaluation development. 

Test design and analysis flow directly from the objectives included 
in the training prototype. At this stage, the focus is on level 1 (re- 
action) and level 2 (learning). The first step is to describe the pur- 
pose of the test, after which a test development committee includ- 
ing both curriculum and evaluation specialists is formed. Before 
items are written, the committee considers the population to be 
tested and the types of items that will be used. Different item for- 
mats (e.g., multiple choice, open-ended) have different strengths 
and weaknesses. For example, multiple-choice items are easily scored 
and more reliable than open-ended questions, but less flexible in 
obtaining subjective, constructed responses. 

After the committee estimates how many items are required to as- 
sess learning outcomes validly, committee members, trainers, and 
other content specialists develop an item pool. Items are matched 
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to training objectives which increases test vaLidity and provides 
assurances that items truly represent content. Evaluation special- 
ists edit items for consistency, validity, and appropriateness. The 
number of items selected per competence reflects the amount of 
training time devoted to that topic. Draft instruments are field- 
tested with persons comparable to those who will be trained. In 
addition to completing the test, these individuals will give unstruc- 
tured feedback regarding the test. Based on this feedback and an 
item analysis, the test is revised and administered to a different 
sample. The recycling process continues until the committee de- 
termines that the test is sufficiently valid and reliable. After training 
begins, test items are scrutinized to determine if they are adequate 
for the assessment. 
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Training 



Figure 4 

Training Phase of MTS 
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Training Phase Continued 



Training is the most publicly visible component of the system. Fig- 
ure 3 summarizes the specific procedures utilized during this phase. 
Pre-training strategies focus on trainee recruitment and selection. 
By describing potential program benefits, trainees are encouraged 
to compLete the program. Trainees review pre-training background 
materials and complete pretests based on these readings and ac- 
tivities. Formal training ends as trainees compLete posttests and 
contract for involvement in job transfer activities with supervisors 
or job mentors. After training, trainees receive reports that describe 
their posttest performance, including suggestions for improving per- 
formance on competencies which they did not master during train- 
ing. 

As training is conducted, formative assessment data are collected 
and this information is fed back into the system to improve and 
refine the program. The process includes formal and informal dis- 
cussions and focus groups conducted by trainers and observations 
by supervisors. Program staff examine this information to determine 
if the actual training procedures match those that were planned. 
They also make appropriate modifications to improve practices that 
are less effective than they anticipated. LeveL 1 (reaction) data are 
collected using Likert items that examine specific aspects of train- 
ing (e.g., trainer enthusiasm, quality of instructional materials). 
LeveL 2 (Learning) assessment is accomplished by pretesting and 
post-testing attitudes, knowledge, and skills primarily with muLtiple- 
choice items. This information is used to prepare individual trainee 
reports described earlier, as well as reports described in the sec- 
tion on summative evaluation. Subjective comments that describe 
trainee reactions are also collected and sent directly to trainers for 
their review. Based on these comments, trainers refine their courses 
appropriately. 

Job transfer is a critical aspect of successful training. Attractive, 
well-received training which does not improve job performance 
wastes time and resources. The model includes pre-training activi- 
ties in which supervisors prepare staff by describing performance 
expectations that will result from training. Post-training activities 
include supervisors who engage staff in performance contracts, re- 
medial training, coaching, mentoring, and formal training which 
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reinforces and rewards improved performance. This feedback maxi- 
mizes the transfer of training and is an essential, component of com- 
petency-based training. Supervisors also compLete related training 
to improve their competence to work with staff. 

Supervisors maximize job transfer by providing staff with psycho- 
logical support and resources that support them as they practice 
newly learned behaviors. Management staff are in the best posi- 
tion to serve as role models and to reward and reinforce compe- 
tent performance. Effective follow-up support during this post-train- 
ing period contributes substantially to successful worker perfor- 
mance. Follow-up activities involve formative and summative evalu- 
ation strategies to determine which activities and job conditions 
promote transfer to performance. 
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Figure 5 

Evaluation Phase of MTS 
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Evaluation Phase Continued 



Figure 4 contains detailed procedures for the Evaluation phase. 
Summative evaluation is based primarily on data from pretests and 
posttests administered by trainers. Tests are developed and vali- 
dated using procedures described earlier. Instruments include items 
on demographics, self-assessment of competence, attitudes towards 
training, and open-ended subjective responses. Before training be- 
gins, each participant completes a pretest which provides baseline 
data regarding performance on cognitive items based on the cur- 
riculum, as well as each person's self-assessment of competence 
in areas related to the training program. After completing train- 
ing, each participant completes a posttest that assesses mastery 
for the total test and for specific content areas. Tests are produced 
using Teieform which creates a form which can be optically scanned 
using a conventional flatbed scanner. Staff scan data into a data- 
base and use Microsoft Access programs to produce nine different 
types of reports. 

One of these reports is a "Trainee Report" produced for each par- 
ticipant. It lists content areas, mastery levels of satisfactory or un- 
satisfactory, and brief messages describing what the score means. 
An unsatisfactory performance level for a content area directs par- 
ticipants to review specific content from training materials distrib- 
uted during training. 

County DSS training directors receive copies of an Agency Report 
that lists agency workers who completed training, shows mastery 
status for each person for each content area, and pretest-posttest 
percentage improvement. Training directors receive copies of train- 
ing materials, recommended assignments for content areas in which 
trainee performance was unsatisfactory, and training suggestions 
for related training activities. This report, in combination with the 
Trainee Reports , enables staff developers and supervisors to develop 
ongoing, individualized training programs to help caseworkers mas- 
ter attitudes, skills, and knowledge required for competent perfor- 
mance. 

Additional reports based on trainee performance are produced for 
trainers, training supervisors, and state-level supervisors. These in- 
clude the following: 
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Trainer reports contain a summary and interpretation of 
trainee comments. These include item analyses that 
describe how well trainees performed on each item. 
Evaluation staff meet with trainers as part of the formative 
evaluation process to improve deficient items and to 
determine if instruction should be modified to improve 
trainee performance. 

Training organization reports summarize performance and 
attitude results for groups of trainees classified by courses 
completed, demographic characteristics, geographic re- 
gions, and agency. Summary reports show cumulative 
achievement of specific training units and trainees. Staff 
use this information for formative and summative evalua- 
tion, as well as for supervisory purposes. 

State-level reports include copies of selected organization 
reports, as well as cumulative reports for similar training 
conducted by different organizations across the state. Level 
3 (behavior) and level 4 (performance) data are collected 
during follow-ups of trainees and their supervisors. These 
follow-ups examine whether training helped staff master 
job-related competencies and whether their performance 
had improved after training. Follow-up, which focuses on 
how well training helped caseworkers perform more effec- 
tively on their jobs, is an integral component of the model. 

Level 4 (results) data are collected during cost-benefit analysis and 
impact assessment. Cost-benefit analysis is "the process of adding 
up the cost, subtracting the costs, and choosing the alternative 
that maximizes the net benefits" (Gramlick, 1990, p. 2). In other 
words, how much money was saved by doing it this way? Assess- 
ing the amount of money saved by conducting training is a diffi- 
cult, but essential component of the system. Unfortunately, the pro- 
cedures involved in conducting cost-benefit studies are often costly 
themselves, and program directors are reluctant or unable to re- 
duce training to fund such studies. However, if training programs 
devoted more money to research during the early stages of train- 
ing until a judgment was made regarding the program effective- 
ness or ineffectiveness, better evidence could be assembled. 
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However, with proper planning it is possible to conduct well-de- 
signed cost-benefit research. Several CDHS studies provide models 
for this type of research. Floss (1990) compared the costs of an 
innovative HMO program and conventional care for Medicaid clients. 
A study in Erie County (McCowan, 1991) showed that clients pre- 
ferred a lower cost managed care program to traditional services 
provided at the county LeveL. 

Impact assessment is the Last step in the system. It focuses on 
transfer of training which is the effect of training on trainee and 
agency performance (Broad & Newstrom, 1992) and corresponds to 
Kirkpatrick's (1994) LeveL 4 evaluation. Data can be collected us- 
ing surveys, focus groups, personaL interviews, telephone interviews, 
and site visits, as well as from muLtipLe sources including local agen- 
cies, summative evaluations, benefit-cost analyses, and impact 
analyses. This information is recycled by management staff during 
the Program Design phase as they pLan new training cycles. Im- 
pact assessment is often limited by the same constraints which af- 
fect cost-benefit studies — namely, cost, time, and data collec- 
tion difficulties, but it is possible to conduct this type of assess- 
ment. 

CDHS conducts semi-annual follow-ups for the total population of 
trainees who completed programs during the preceding six month 
period using scannable Teleform instruments. In 1997 over 20,000 
trainees and their supervisors were involved in these surveys. Each 
follow-up form contains the trainee's name, the training program 
compLeted, and dates of training, in addition to the following three 
questions rated on a 5-point scaLe ranging from "strongly disagree" 
to "strongly agree." 

"I learned specific, job-reLated skills and knowLedge." 

"Since completing the training, I have used the skills and 

knowLedge on the job." 

"This training helped me do my job better." 

The survey includes from three to five additional questions based 
on specific competencies on which training was based. Trainees in- 
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dicate whether they used these behaviors effectively as a result of 
completing training, and their supervisors complete a similar form 
to rate the trainees on the same competencies. For example, the 
following items were used for the FosterParentScope: Train the 
Trainer program: 

"Have you used these behaviors effectively as a result of 
completing the training?" 

"Use FosterParentScope materials to develop training 
programs." 

"Develop workshops with consultants using FosterParentScope 
materials." 

Trainees also indicate whether they would change, discontinue, 
revise, or offer the training in exactly the same format. They also 
complete the following open-ended questions: 

"In what ways did your agency support your use of these 
behaviors?" 

"What obstacles prevented you from using these behaviors? 

"Please make any additional comments you feel are 
appropriate." 

Respondents mail the forms or fax them directly into a database, 
and CDHS produces reports using Microsoft Access programs. 

CDHS has completed other impact assessments. McCowan and 
Wegenast (1989) used a Solomon 4-group pretest-posttest design 
to show that training had a significant, positive effect on the ability 
of caseworkers to identify and write specific objectives. Other re- 
search projects designed by CDHS will be completed in 1998. One 
study compared the effects of two different spend-down programs 
on low-income clients in two similar, contiguous counties. Another 
study used a time-series design to compare past, current, and fu- 
ture employment data to assess the impact of an employment ini- 
tiative for clients on public assistance. A third study, described ear- 
lier, examined the impact of training on the performance of 400 
New York City risk assessment workers. 
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The CDHS training management system is a set of practical, sys- 
tematic procedures designed to manage and evaluate training pro- 
grams. It is a complete model that meets the necessary criteria for 
competency-based training, namely: 

Clear job descriptions and program outcomes. 

Needs assessments based on job-related competencies. 

Structured hierarchy of domains, competencies, and objectives. 

Posttest assessment of trainee performance compared with 
clear criteria. 

Remedial training and OJT mentoring to assure trainee mastery 
of essential material. 

The system conforms to the criteria required for a system as de- 
scribed earlier in this paper. It integrates the complex procedures 
required to develop, deliver, and evaluate competency-based train- 
ing. It provides a general model for a wide variety of different train- 
ing and educational programs in the public and private sectors. 

The system clearly communicates how its goals are achieved and 
clarifies the relationships that exist among goals, procedures, and 
outcomes. It describes how assessment measures are designed and 
utilized and provides an iterative cycle of development, testing, and 
redesign. It includes a full array of computer-supported systems 
to manage and report training data, ranging from the scheduling 
and planning of training through evaluation and follow-up at the 
reaction, learning, performance and results levels. Finally, the ef- 
ficacy of the system has been tested and refined over a 20-year 
period in a large, statewide social services training program. Its pro- 
cedures could readily be adapted to manage a broad spectrum of 
training programs both in the public and private sector. 
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